Adding atunnel - networking component within ateom - #559
Conversation
| extracts the Actor name and Atespace from the `Host` header and calls the | ||
| Control Plane to resume the Actor and resolve its current worker assignment. | ||
|
|
||
| * **Worker Tunnel**: After resolving the assignment, `atenet-router` opens an |
There was a problem hiding this comment.
Change this to Worker Ingress Tunnel
| extracts the Actor name and Atespace from the `Host` header and calls the | ||
| Control Plane to resume the Actor and resolve its current worker assignment. | ||
|
|
||
| * **Worker Tunnel**: After resolving the assignment, `atenet-router` opens an |
There was a problem hiding this comment.
Worthy adding a one-line like egress redirection through atunnel is implemented but not yet enabled, which would stop the readers from concluding either that it's live or that it doesn't exist.
There was a problem hiding this comment.
Review that before we merge - this is ai generated fully
There was a problem hiding this comment.
text changes look ok to me
a0c14f2 to
6030db8
Compare
| signerName: podidentity.podcert.ate.dev/identity | ||
| labelSelector: | ||
| matchLabels: | ||
| podcert.ate.dev/canarying: live |
There was a problem hiding this comment.
TODO: not sure if we need this or not. Discuss with other community members
| } | ||
|
|
||
| // The actor is reached through the in-worker atunnel ingress server, which | ||
| // listens on :443 (mTLS) and forwards to the actor's :80. The worker no |
There was a problem hiding this comment.
I'm thinking we should use not 443, but own port just in case we need the standard HTTPS port later.
There was a problem hiding this comment.
Discussed offline but commeting for posterity. This port is standard https.
When we introduce CONNECT on Ingress (client -> atenet/pluggable ingress) - we may also have connect to atunnel, and in that case, we can consider having that a 8443 or any other special port.
| return nil | ||
| } | ||
|
|
||
| func runAtunnel(ctx context.Context, upstream *url.URL) (*atunnel.Server, *atunnel.Egress, uint16, error) { |
There was a problem hiding this comment.
is this going to be same or dupe for ateom-uvm
| func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.CheckpointWorkloadRequest) (*ateompb.CheckpointWorkloadResponse, error) { | ||
| s.lock.Lock() | ||
| defer s.lock.Unlock() | ||
| if err := s.deactivateActorNetworking(ctx); err != nil { |
There was a problem hiding this comment.
is this just a bug that it's missing on the CheckpointWorkload path? Specifically, we should just make this change outside of this change.
There was a problem hiding this comment.
deactivateActorNetworking is a new function this PR added. Ideally we are trying to stop admitting traffic and drain active streams before the Actor network is torn down. Since (1) we introduced atunnel and it takes care of all networking connections for the actor and (2) CheckpointWorkload suspends the actor. I think the best home for this addition is within this PR.
| } | ||
|
|
||
| // Client opens actor egress streams through an mTLS-authenticated gateway. | ||
| type Client struct { |
There was a problem hiding this comment.
nit: would be good to make this more consistent, calling everything egress related egress vs a very generic sounding Client.
There was a problem hiding this comment.
I originally did it and then reverted, there is nothing egress-only in client. Client implements EgressDialer though.. Suggest we leave it this way
b37233c to
e253ee0
Compare
Benjamin Elder (BenTheElder)
left a comment
There was a problem hiding this comment.
One Plausible AI review finding.
| return loadCredentialBundle(s.credentialBundlePath) | ||
| }, | ||
| ClientAuth: tls.RequireAndVerifyClientCert, | ||
| ClientCAs: clientCAs, |
There was a problem hiding this comment.
🤖 should-fix 🟡 – The client CA pool is frozen at process start while the leaf beside it reloads per connection. GetCertificate above re-reads the credential bundle on every handshake, with a comment explaining that this is what lets kubelet's rotation take effect — but clientCAs comes from a single os.ReadFile in NewServer.
Both files live in the same projected volume: workerpool_apply.go gives it a podCertificate source for credential-bundle.pem and a clusterTrustBundle source for trust-bundle.pem. A ClusterTrustBundle projection exists precisely so kubelet can push CA changes into a running pod, and ateom outlives many activations, so after a signer CA rotation this server keeps verifying against the old pool and rejects the router until the worker pod restarts.
GetConfigForClient is the usual fix: return a *tls.Config per connection with a freshly built ClientCAs, mirroring what GetCertificate already does for the leaf. client.go reads its RootCAs once too, though NewClient runs per activation, so the window there is much shorter.
There was a problem hiding this comment.
Lior can you take a TODO to follow up on this
There was a problem hiding this comment.
added a todo.. do we have rotation today?
There was a problem hiding this comment.
yes, we should have it...otherwise substrate will stop working after 1 - 2 days due to the certs having very quick expiry.
There was a problem hiding this comment.
sg, added a todo and will open an issue for it as well. Rebased now, hopefully all should be good to go
aa4874d to
65c5e58
Compare
atunnel is the in-worker component that carries Actor traffic in both
directions over authenticated channels.
* Server: an mTLS HTTPS listener that terminates connections from the
ingress gateway, authorizes the peer by its SPIFFE identity, checks
that the request targets the Actor currently activated on this
worker, and reverse-proxies to the Actor over the private veth.
* Egress: an activation-aware TCP proxy for transparently intercepted
Actor connections. It resolves the original destination via
SO_ORIGINAL_DST and forwards through an EgressDialer, asserting the
Actor identity to the far end. Dormant until an egress gateway is
configured; the client and dialer land here so the package is
reviewed as one unit.
Activate/Deactivate bracket an Actor activation so traffic is only
carried while an Actor is assigned to the worker, and Deactivate drains
in-flight streams before the Actor network is torn down.
Every worker pod now hosts an atunnel ingress server on :443 and an
atunnel egress listener, both long-lived, with per-activation
Activate/Deactivate bracketing every Run, Restore and Checkpoint.
The nftables rules ateom installs change accordingly:
* The pod-IP:80 -> actor-veth:80 DNAT is gone. Worker port 80 is no
longer an Actor ingress path; the only way in is the mTLS listener
on :443, which authorizes the caller and checks the Actor is the one
currently assigned to this worker.
* A prerouting REDIRECT sends Actor TCP egress to the local atunnel
egress listener, preserving SO_ORIGINAL_DST. It is only installed
when the activation carries an egress gateway address, which nothing
populates yet, so the masquerade path is unchanged and Actor egress
behaves exactly as before.
The ateom Run/Restore protos gain the two fields that arm that path:
egress_gateway_address, which decides whether the redirect is installed
at all, and actor_version, the Actor resource version ate-api observed
when assigning the worker, which atunnel asserts to the egress gateway
as a lower bound on trustworthy Actor metadata. Both are consumed here
and left unset. atelet and ate-api start populating them in the egress
gateway change, which is the point at which they mean anything, so this
change adds no new requirement to the atelet wire contract.
atecontroller gives worker pods the podidentity credential + trust
bundles (the atunnel server identity), the servicedns trust bundle, and
container port 443. podidentitysigner now issues certs with
ExtKeyUsageServerAuth as well as ClientAuth, without which the worker
cannot present its podidentity cert as a TLS server cert and the
gateway handshake fails.
The router used to rewrite :authority to the actor's worker pod IP and let the dynamic_forward_proxy cluster resolve it, reaching the actor over plaintext pod-IP:80. The worker no longer DNATs pod-IP:80 to the actor, so that path is gone. Instead the ext_proc resolves the actor to its worker's atunnel ingress address (IP:443) and puts it in x-ate-original-dst. An ORIGINAL_DST cluster dials exactly that address, which leaves the request Host as the actor DNS name -- atunnel needs it to authorize the active actor. The header is set with OVERWRITE_IF_EXISTS_OR_ADD so a client-supplied value can never influence the address Envoy dials. The upstream hop is mTLS: the cluster presents the router's podidentity credential bundle as its client cert and validates the atunnel server against the podidentity trust bundle. Validation matches the SPIFFE URI SAN prefix rather than the dialed pod IP, because the atunnel cert carries only a spiffe:// URI SAN and Envoy's default SAN check against an ephemeral pod IP would never match. atunnel in turn only accepts spiffe://cluster.local/ns/ate-system/sa/atenet-router. The dynamic_forward_proxy cluster, HTTP filter and DNS cache config are removed along with the :authority rewrite.
Bring the architecture and threat-model docs in line with the atunnel ingress path: the router no longer rewrites :authority to a worker pod IP and forwards over plaintext port 80, it opens an mTLS tunnel to atunnel on worker port 443, which forwards to the Actor over its private veth. Drop cmd/atenet/atenet-diagram.png. It predates the ext_proc/ORIGINAL_DST design and is now wrong in the part that matters most. The README section it illustrated gains a short accurate note about the upstream hop instead of a dangling image reference. Add an e2e suite covering the change end to end: TestActorDirectAccess asserts that the worker pod's port 80 is no longer a reachable Actor ingress path (the DNAT rule is gone) and that the same Actor still answers /readyz through atenet-router over the atunnel mTLS hop. It uses the counter demo as its fixture, so it only needs --deploy-demo-counter. internal/e2e/testmain.go picks up ParseSkippedFlags so that `go test` flags (-run, -v, ...) survive pflag parsing and reach the suite.
508de39 to
f2a12d2
Compare
[Will rebase after its lgtm]
Introducing atunnel, a component that sits within ateom and handles all inbound and outbound traffic to the actor. With this change in place there is no direct traffic outside and inside the actor with the exception of DNS and non-TCP which traffic which is still TBD.
this PR;
Huge part of it uses and is built-upon John Howard (@howardjohn) 's work on #460
More specific notes:
New internal/atunnel package — an mTLS ingress server plus a transparently-intercepted egress TCP proxy. Only the ingress half is live in this PR.
Every worker pod hosts an atunnel ingress server on :443. The server authorizes the caller and checks the Actor is the one currently assigned to that worker.
atenet-router dials the tunnel instead of the Actor. The dynamic_forward_proxy cluster is replaced by an ORIGINAL_DST cluster driven by an ext_proc-set x-ate-original-dst header, with upstream mTLS and SPIFFE URI SAN validation. :authority is left untouched, so atunnel can authorize on the original Host.
The pod-IP:80 → actor-veth:80 DNAT is removed. Worker port 80 is no longer an ingress path; the mTLS listener is the only way in.
The egress proxy, the nftables REDIRECT rule and the two new ateompb fields are all gated on an egress gateway address that nothing populates yet, so Actor egress still uses the existing masquerade, unchanged. A pluggable-egress PR is a fast follow.
Tests pass
Appropriate changes to documentation are included in the PR